What is Exadata?
Well I’ve been holed up playing with an Exadata V2 machine for the past several weeks. Wow. Very interesting technology.
I must say that I believe the concept of offloading SQL processing to the storage layer is a game changer and I wouldn’t be at all surprised to see this as a standard feature a few years from now. What that means for other storage vendors is unclear at this point. So for this first post on the topic let me just describe the configuration (and some potential upgrades).
The basic architecture consists of a set of database severs and a set of storage servers.
Database Servers:
- Sun x4170 (1RU 64x server)
- 2 – Quad-core Intel Xeon E5540 2.53GHz processors
- 72G Ram (18x4G Dimms – max of 144G using 8G DIMMs)
- Dual-Port QDR InfiniBand Host Channel Adapter
- HBA with 512MB Battery Backed Write Cache (only for internal disks???)
- 4 – 146G internal drives (SAS 10,000 RPM)
- dual hot swappable power supplies
- no spare/empty slots!
Here’s what the Database Servers look like:
Storage Servers:
- Sun x4275 (2RU 64x server)
- 2 – Quad-core Intel Xeon E5540 (2.53GHz) processors
- 24G Ram
- Dual-Port QDR InfiniBand Host Channel Adapter
- HBA with 512MB Battery Backed Write Cache (only for internal disks???)
- dual hot swappable power supplies
- 4 – 96G Sun Flash PCIe Cards (total of 384 GB)
- 12 – 600 GB 15,000 RPM SAS or 2 TB 7,200 RPM SATA
Here’s what the Storage Servers look like:
In addition to these two components there is are 2 Infiniband switches (redundant) to tie the whole thing together. There is also a KVM and a couple of Cisco 4948 switches. And the whole thing comes in a pretty Rack with a fancy Exadata Logo. Here’s a couple of real pictures taken by a real crew member:
Oracle is big on the balanced configuration. That is to say, they have spent a lot of time trying to eliminate bottlenecks in any one component. That said, it seems to me that at some point they will start to allow more customizable configurations. For example, 72G on the database servers is not a huge amount of memory these days, especially for a consolidation platform that will house several instances. So here are some thoughts on potential ala-cart upgrades that may be possible at some point.
- The database servers are capable of using any Intel 5500 series chips. They are shipped with 2.53GHz 5540’s which are middle of the road for the series. The 5540’s could potentially be replaced with 5570’s which are clocked at 2.93 GHz.
- The database servers come with 18 4G Dimms for a total of 72G of RAM. 8G Dimms are available which would increase the RAM to 144G.
- The database servers (Sun 4170) have no spare slots, making it difficult to attach external devices such as tape backup units and other fiber based storage. Swapping the 1U 4170’s for the 2U 4275’s would provide additional slots allowing more connection options.
- Adding additional storage servers is possible regardless of which option is chosen (quarter rack, half rack, etc…) Oracle has already agreed to this as long as they are not mounted in the Exadata rack enclosure.
- External database servers can be configured to access the storage servers. This would require Infiniband cards on the external servers and of course you’d need to be running 11gR2 to take advantage of the storage servers “special sauce”.
Here are a couple of pertinent statements regarding upgrades out of the Oracle Exadata FAQ:
Can the Sun Oracle Exadata Storage Server hardware or software be modified or customized?
Customization or modification of the Sun Oracle Exadata Server Hardware or the Oracle Exadata Server Software is not allowed or supported. If a customer modifies, customizes, or installs additional hardware or software on the Sun Oracle Exadata Storage Server, the Sun Oracle Exadata Storage Server will no longer be supported.
What additional software can be installed on the database servers in a Sun Oracle Database Machine?
For optimal performance and supportability, Oracle recommends only the Oracle Database and database options, be installed on the Sun Oracle Database Machine. Other than the Oracle Database software we recommend that only agent software, for services like backup or system management, be installed on the Sun Oracle Database Machine.
What are the consequences of making Sun Oracle Database Machine hardware modifications?
Oracle does not currently allow changes to the hardware in the Sun Oracle Database Machine including re-racking of the servers and switches. The Sun Oracle Database machine is a standard configuration that is assembled, tested, and supported as a unit. For many years, Oracle has supported configurations in the market in which the customer can choose any hardware platform and configuration that they desire. With the Sun Oracle Database machine Oracle is taking a different approach and standardizing the complete configuration. We believe a standard configuration will allow customers to achieve stable, high-performance production deployments much faster and with lower cost.
Can a customer add additional storage cells, to an existing Quarter Rack or Half Rack Sun Oracle Database Machine?
Additional Exadata cells should be added to a Database Machine by installing them in a separate adjacent rack. You should not attempt to add Exadata cells to a Half or Quarter rack as it will change the environmental characteristics of those Database Machines and prevent use of the standard Database Machine upgrades.
Can the Oracle Database use Exadata Storage simultaneously with traditional non-Exadata storage products?
Yes, databases can be concurrently deployed on Exadata storage and non-Exadata storage. Tablespaces can exist on Exadata storage, non-Exadata storage, or a combination of the two and this is transparent to database applications. However, Exadata Smart Scan offload processing is enabled only if the entire tablespace being queried resides on Exadata storage.
Can the Database Machine hardware be modified or customized? For example, can I add more Ethernet Cards or Fibre Channel HBAs?
The Sun Oracle Database Machine hardware cannot be modified or customized. There is one exception to this. The only hardware modification to the Database Machine that is allowed is to the administrative 48-port Cisco 4948 Gigabit Ethernet switch is included in the Database Machine. Customers may choose to:
- Replace the Gigabit Ethernet switch, at their own expense, with an equivalent 1U 48-port Gigabit Ethernet switch that conforms to their internal datacenter network standards. This replacement must be performed by the customer after delivery of the Database Machine at their expense and labor. If the customer chooses to make this change, Sun and Oracle cannot make or assist with this change given the numerous possible scenarios involved and because it is not scoped as part of the standard installation. The customer must supply the replacement hardware and make or arrange for this change through other means.
- Remove the CAT5 cables connected to the Cisco 4948 Ethernet switch and connect them to the customer’s network through an external switch or patch panel. The customer must perform these changes at their expense and labor. In this case, the Cisco 4948 Ethernet switch in the rack can be turned off and left unconnected to the data center network.
Well, that’s it for now. I’ll try to post soon on some of the more interesting features of the platform. Let me know what you think or if you have any specific things you’d like me to investigate.
“The database servers (Sun 4170) have no spare slots, making it difficult to attach external devices such as tape backup units and other fiber based storage. Swapping the 1U 4170’s for the 2U 4275’s would provide additional slots allowing more connection options.”
…you must be working with a customer that has either a half or a quarter-rack. Thoughts of larger form factor hosts would seem feasible in that case I suppose. A full-rack configuration has no spare room to “fatten” the hosts.
…glad to see you’re getting to have some fun. Fast stuff, eh ?
Kevin,
Yes – 2 quarter racks. I was thinking Flash Cache in the DB Server might be interesting as well. Yes, screaming fast on the big tables!
What’s with the picture by the way? My eyes aren’t that great anymore but it looks like a jeep with a deer blind built on the back.
Kerry
Kerry,
Flash cache in the Database Machine hosts would be of no value… trust me… I know… in fact, if you had joined those other 6 attendees in my Hotsos Symposium 2010 session you’d know why 🙂 Hold it, you can get the slides since you were a speaker…
I plan to do a contrast presentation webcast-style on the topic of these two different Oracle Flash technology offerings. There are a lot of DIY types that seem to suppose that flash is flash so they can back the “upstairs” with flash and dangle regular old spinning rust for the downstairs and voila the product is a self-made Database Machine. The problem is, it isn’t so.
That avatar is a photo of a friend’s Jeep circa ’49 or so. It never moves. We hang our deer in the same building though when successful on a hunt if that’s any consolation 🙂
Thanks a lot for taking this hot topic Exadata V2.
Will be more usefull if you could provide some insights with test cases on advantage of using Exadata V2 compared to absence of this technology.
Also some basic stuff with examples will more helpfull as some DBA’s are not so lucky to work on this technology.
-Yasser
Yasser:
The main advantages of Exadata are #1) database performance and #2) ease of deployment. The Oracle Database Machine comes from the factory as a pre-built system of servers, storage and networking. Oracle installs & configures the machine, and the system goes from the loading dock to loading data in about 2-3 days. The system performs better than traditional hardware architectures because the Oracle software is tightly integrated into the hardware. Part of the RDBMS code actually runs INSIDE the storage tier. The software inside the storage knows how to run fragments of SQL statements, and it knows how to utilize the built-in cache to best advantage for database performance. Oracle software doesn’t run inside of a traditional disk array, so you simply cannot get this degree of integration without Exadata.
-Hope this helps…
Chris
I envy you, Kerry!
This is some stuff I have been itching to put my hands on.
It might be great if you can test both DW queries ( that are great for offloading) and some ETL processes performance. I would be delighted to see cell offloading in work, execution plan differences, statistics differences etc.
In the non-exadata configurations, we use consistent gets, db block gets, db time etc to resolve performance issues. How is that going to change with exadata? how effective are the tools that we currently use, in exdata environment?
Lots of questions, so little time !
Kevin dude,
you are humble and you had lot more than 6 attendees in your hotsos presentation. Good stuff, that you present.
Cheers
Riyaj
Hi Riyaj,
Yes, I feel very fortunate to be able to work on that system. There are stats for seeing some of the unique features such as offloading (how much data is actually transferred, etc..) and what’s returned from flash cache vs. actual disk access. Although the stats are not quite as easily accessible as we are used to for non-Exadata stuff (i.e. not all of it is accessible at the statement/cursor level via V$SQL). But there are new wait events, new plan options (which are available via xplan, etc…) I intend to share that info as soon as I have some time to put it together. On a side note, we have our own Exadata showing up in a couple of weeks. So we should be able to do whatever kind of testing we want, without having to worry about “real” work being done on the platform as well. That should be fun! Although there is no substitute for being able to look at a real production system, so as I said earlier, I am really thankful that we have had the opportunity to work on the real system (which is in limited production now by the way).
Kerry
[…] Oracle Whitepapers / Presentations; Fun with Exadata […]